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FOREWORD 

Vocationai education staffs in thp states are making a concerted" effort to make evaluation 
, effective in stimulating program improvemen.. Obviously a multitude of problems surface when 
new activities start. A$ a part of the evaluation function of the National Center, technical evaluation 
assistance was provided to four states (California, Colorado, Maine, and Alabama). The problems 
addressed in this report grew out of this project supplemented by observation, copsultatioh, and 
input from many other states. This report attempts to assist states in dealing with these problems 
by presenting alternative solutidns with some related advantages and disadvantages as well as 
observations and recommendations on each problem. 

The National Center is particularly indebted to Bill Stevenson who authored this report 
and to Marion Franken and Eliseo Ponce for their thoughtful reviews and suggestions. Contributions 
to the report were made 1- - N. L. McCaslin, associate director, and F. L. McKinney, program 
director of the Evaluation and Policy Division. 
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National Center and to the evaluation coordinators in each of the states wiro contributed to this 
report. Particular thanks ^re expressed tb John Klit, Illinois, and Ralph RosJ, Oklahoma, for their 
thorough and insightful review of the draft of this document. Their comments and suggestions 
have been carefully considered in revising'this report. 

A special note of appreciation is extended to Nancy Powell, secretary, for her patient and 
skilled assistance. Finally, appreciation is extended to the Bureau of Occupational and Adult, 
Education, U.S. Office of Education for their support of this effort through the contract for the 
National Center for Research- in Vocational Educati6n, 
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PREFACE 



Communication is recognized as critical to the effective operation of an organizatipn. State 
vocational leaders are seeking to improve communication m order to increase the impact of 
evaluation and other activities on program improvement. Following are some thoughts on commu- 
riicatrbn which might be useful in increasing this essential phenomenon. 



Communication 

Communication thrives where there is - . 

mutual respect, > 
♦ 

absence of fe^r or jealousy, 

a contribution to make, 

mutual acceptance of objectives. 
Communication is stimulated by 
^ close physical proximity, 

example and encouragement by leaders, 

recognition and reward for effective comm mication. 
Communication ceases when there is 

fear or lntense cpmjietition, 

exclusive rights to information, 

restriction of information by leaders, 
^ loss of trust, 

lack of mutual support, ^ ^ 
"doubt about accuracy and reliability of others. 
People then become 

protective of their own, 
guarded with others, 
unknowing and uncaring. 

^ William W.StevenVon 
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CHAPTEI^I 
INTROC^UCTION 



An eminent philosopher has attempted to divide our heeds and the search for sojutions into 
three levels: predicaments, problems, and puzzles. By his definition, a predicament is said tcbe a 
situation we must get ourselves out of;isi problem, a grouping of predicaments arranged for study or 
research; and a puzzle, games played by- a few scientists with ideas having ho apparent application. 
■|,n light of this definition, what is pc^sented in this report would appear to be a set of predicaments. 
However, in light of general usage, the ^reas of need wil l be termed problems. Several of these 
problems are opierational or procedural, and others are content-oriented. The only, criteria used in - 
selecting these problems were that (1 ) they represent a situation that should be remedied or a rtee^ 
that must be met, and (2) th§ problem was found to be common to a number of states attempting 
to develop or improve their^Sy«tem of vocational education evaluation. 

In no instance will all of the alternative solutions be applicable to any one state; however, this 
report attempts to ekpand the range of alternatives considered. In some instances the disadvantages 
presented may become advantages when the difficulties have been faced and surmounted. 

This report dpes not provide a single solution to a problem; rather, it suggests several possible 
approaches a state should consider in seeking methods of jmpi'ovement. In too nhany, instances, 
decision-makers limit themselves more by the numbe^ and variety of alternatives considered than by 
their ability to adopt the best solution. When presented witli a problem, our minds are inclined to 
automatically jump to a solution. We then begin the process of working out the details of imple- 
menting that single solution. an example, when faced with the necessity to collect information 
from employers of vocational educafion completers, we instantly think, "questionnaire," and the 
rest of our thinking concentrates on|how to develop, distribute, and retrieve the information- 
gathering instrument. In fact, a mailed questionnaire is not the only-and perhaps not even the 
best— way of getting the information we want from employers. 

Another important question we should pose in seeking alternatives is "What do we want this 
process to tell us and for what purpose?" It is hoped that some justification beyond legislative or 
reporting requirements can be found to guide us in more efficiently and ef/ectively carrying out 
our responsibilities. The alternatives presented in this report should be viewed not as a complete 
listing of all possible choices, but as a starting point for staff discussion and exploration. In studying 
the selected problems in this refDort, the reader should try to avoid the ready solution and, instead, 
search these pages and his/her own mind for innovative approaches to dWp-r.i evaluation to more 
completely accomplish the mission. 
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CHAPTER II 



PROBLEMS, ALTERNATIVE SOLUTIONS, 
RECOMMENDATIONS, AND OBSERVATIONS 



Problem 1: To expand representation in evaluation j| 

In too many instances, state vocational evaluation programs are planned, developed, operated, 
and evaluated only by individuals who have a direct interest in perpetuating the present system. 
This situation he '-Is the potential for a limited viewpoint and restriction of alternatives considered. 
More diverse backgrounds would enable panel members to view needed changes froiin different 
perspectives and expand the list of possible alternative solutions. When individuals to participate in 
evaluation are being selected, inclusion of representatives of special populations enrolled in pro- ■ 
grams or in need of training should be considered. Such a pahel could provide a view of programs 
and needs that might be quite different from the commonly expressed opinions. 

Another potential trouble spot that might be alleviated by increased participation is the lack 
of .knowledge about vocational education on the part of the general public. Involvement in anv ' ' 
phase of the evaluation process can expose individuals outside the system to both the needs and 
benefits of vocational education, exposure which can only lead to stronger support for worthwhile 
programs. Each state should carefully examine who is evaluating with an eye to achieving the 
participation of a diverse group, of individuals representative of a broad segment of tne society; 

% ' ' . ■ . 

Alternative 1 : Change the makeup of teams to assure at least one outside teacher and one advisory 
committee member (employer) in each subject matter area. Additionally, wherever feasible, ingjude 
a counselor, a nonvocational teacher, a teacher educator, an administrator, and a parent. 

Advantages: 

• Increased potential for seeing problems ijnd finding viable solutions 

} ' 

• Better understanding of content areas arid teaching methods provided by teachers 
and teacher educators 

• Setter methods of achieving student employability provided by employers 

• Appreciation of the values of vocational programs acquired by participating 
nonvocational teachers 

• Increased community understanding and support of good vocational programs 

• Formation and use of advisory committees supported 
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Disadvantages: ^ 

• Possible change in policy required 

• Limited increased cost 

• Difficulty in orienting, organizing, and scheduling visiting teams 

/* 

Alternative 2; Expand the number and type of individuals interviewed by the team to include ^ 
at least one former student, a parent, and a counselor or other nonvocational school representative* 

Advantages: ^ ' 

• Feedback provided to teachers on reactions of students interviewed 

• Feeling of inclusion acquired by those asked to comment 

• Alternative Views and steps toward improvement result from diverse interviews 

t A comprehensive view of program activities and services acquired by nonvocational 
individuals 

Disadvantages: 

• Additional time required for expanded interviews 

• The evaluation report and recommended solutions still the /ork of only a limited 
number of individuals 

• Limited community exposgre to vocational education 

• Increased coordination problems 

• Reluctance to discuss problems with "outsiders" 

Alternative 3: Include broad representation in a locally developed self-evaluation. 
Advantages: 

• Community representation in the evaluation process 

! 

• Increased contact between vocational educators and employers 

• Strong support for recommendations » 

Disadvantages: 

• Decisions suspect when made by vocational personnel regarding the validity of views 
obtained from interviews 

• Contact between the evaluation team and community limited to school personnel 

• Increased chance for conflict between local board and community members 

« 
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Racommendatinns and Observations 

■• ■ 

A comblnalion of the above and other alternative., should be considered. When the local 
school begins t6 realizef the benefit? of expanded participation and when fear of evaluation 
diminiaihes, the movement is likely to develop a mornefifum of its own It is recommended that the 
state set minimums for representation on review teams and for self-evaluation involvement. The 
very le^st should be empjoyer participation with inclusion, wherever possible, of teachers (voca- 
tions! and nonvocatirnal), parents, students, and' representatives of the business communit/ and 
teacher-training institutions. 



Results of the implementation of one or a combination of alternatives should be assessed 
during the schpol year. To the extent possible, results should be recorded in the form of (1) number 
of employers, teacher educators, and teachers serving on teams; (2) number of recommendations 
made by these individuals included in the report; and (3) perceptions of team leaders and teachers 
of caluated programs i elative to the contributions of these individuals. 
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Problem 2: To increase the effectiveness of ^the supervisory (subject matter specialist's) review 

Closely related to Problem 1 (involvement of diverse individuals) is the problem of evaluation 
by only the program supervisor, since evaluations made by the individual responsible for supervision 
of a program are likely to suspect. Although the supervisor may be very knowledgeable about 
progrr.m weaknesses and needs, judgrnents about the quality of an operation when made by a 
person closely involved in that operation are subject to 'Skepticism. In order to avoid putting super- 
visors in an unteYiable position, it is important that every effort be made to assure objectivity and 
credibility. The following alternatives represent the minimum effort required to broaden ihe base 
of evaluation, 

Of 

Alternative 1: Involve at least one member of the local advisory committee in the program review. 
Adva^^ag^s: 

• Minimal additional expense 

• Increased undf^rstanding on the part of advisory committee members of their roh 
in the program / ^ 

• Strengthening of recjbmmendations made for improvement 

• Stimulus'provided for the fcrm'^tion and use of advisory committees 

• At least one outside view of program quality and needs provided . 

Disadvantages; 

• Careful scheduling of supervisory visits required 

• Potential for bias 

Alternative 2: Provide supervisor with as much supportive data on the program as possible. 
Advantages: 

• Data system informed of data needs 

• Student employmert success record for each program provided to the supervisor 

• Results ot program changes observed by the supervisor 

• Possibility of process/outcomes study provided 

Disadvantages: 

• Possible data system modifications required 

• Special preparation for supervisory visit required 




Alternative 3: Make basis for judging program quality as objective as possible by developing 
measurabie^tandards and improved instrumentation. 



Advantages:' / 

^ Decreased pressure on supervisors to point cut program deficiencies 

• Increased effectiveness of evaluation 

• increased credibility of evaluation 

Supervisor provided with better basis for recomnriendations 
Disadvantage: 

• Embarrassment to tfie local community resulting from identified, program weaknesses 

Recommendations and Observations 

Supervisors should be provided with all applicable data available In preparation for a school 
visit, including long-range enrollments of regular and special populations, student and employer 
follow-up data, as well as conclusions.of self-evaluations am"* response to previous recommendations. 

^ It is recommended that at least one member from each program advisory committee be 
included in each program reviewed by the superv sor. This policy ^A/ill present an<^her perspective, 
increase credibilitY^pf the evaluation, and provide for additional involvement of the advisory 
committee. C 

lnstrumen>^s used by supervisors and those assisting them should, to the extent possible, 
measure objective program Standards in terms which avoid supervisor judgment. Fop example, it 
may be more appropriate to ask the number of visits a teacher has made to employers rather than 
to ask whether adequate employer contacts have been made. 

This, as the total method of evaluation, at best has many limitations. States should carefully 
consider ways in which a more objective and creditable view of program quality (in the opinicn 
of those outside vxational education) can be achieved. 
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Problems: To follow through on evaluation recommendations 

Probably the most important function of an effective evaluation team is not to evaluate but 
to make recommendations for improvement The recommendations resulting from a self-evaluation 
by local teachers and administrators are also very important. Schools^ne6d assistance/ not only in 
identifying problems, but in solving them. A foilow-throggh system on the recomm'endations is 
essential if maximum benefit is to be realized from the evaluation procedure. 

(^ollow-throu'gh consist^ of several important phases or the pisrt of different agencies. First, 
the state should be in a position to provide assistance in implementing recommendations. Second, 
the state should require that critical program improvements be made if programs are to continue. 
Third, there should be feedback to the evaluation coordinator on how recommendations have been 
dealt with. It is important that local and state staff clearly understand their areas of responsibility " 
in this effort to help schools and programs provide better vocational education. (See following 
figure.) ■ . . 

The problem seems to stem from the fact that^in many states evaluation guidelines,are not 
clear on at least two points: ( 1 ) Who at the state level is responsible for working with local 
administrators and teachers to see that recommendations of the evaluation team are carried out? 
,(2) What mechanisms are to be used to let state planners and evaluators know that recommendations 
have been implemented and what the results of that implementation have been? Communicating 
the recommendations of the evaluation team to those responsible for their implementation at both 
the state and local level is also a part of this problem. 

A part of the solution appears to be a matter of fixing responsibility for certain phases of 
the evaluation function at the state level. This is a matter of determining who can do what most 
effectively. It is important in assigning responsibility to specify how and to whom these actions 
and the ensuing results shall be reported and also indicate who will act on an assignment. Linejs of 
responsibility for action and communication must be established. These assignments should made 
by someone beyond the program level (such as the state director or an assistant) in order to ensure 
compliance. 



Alternative 1 : Give staff members who serve as subject matter specialists the principal responsibility 
for car: y ing out the recommendations. ' 

Advantages: 

• Knowledge of programs and latest trends in subject matter areas provided 

• Increased credibility of advice 

• Contacts with other sources of assistance provided 
Disadvantages: 

• Possible reluc nee of subject matter specialists to call upon others for assistance outside 
their own areas (administration, curriculum, guidance, etc.) 

• Possible reluctance of individuals to criticize their colleagues and to promote change 

• Preference on the part of staff members to maintain the status quo 
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Alternative 2: Make state administrative^ spacialists from education department available to assist 
with special needs of schools. 

Advantages: 

• Availability of personnel trained to assist in irnproving administration for ^ 
vocational education 

• Interaction between state administrators and local schools 

• Close ties befjveen local administrators and state administrators used to advantage 
Disadvantages: 

• Difficulty in scheduling -the activity 

• Possible limited ability to communicate improved management practices 

Alternative 3: Make staff working in support semcesrifor curriculum, finance, special needs, etc. 
available to assist with problems related to their area ci^ expertise. 

Advantages: 

• Specialists available to local schools 

• Assistance of these specialists formalized 

• Areas of greatest need for support services work ide-ntified 

• Help directed where it is most needed as indicated by evaluation recommendations 
» State staff become more attuned to local needs 

Disadvantages: 

• Difficulties in scheduling the required time 

• Possible resentment and fear on the part of teachers from what they regard as 
outside interference 

Alternative 4: Make teacher educators aware of the needs and ask them to assist teachers with 
solutions to their problems. 

Advantages: 

• increased understanding of local programs on the part of teacher educators 

• Preservice and inservice need? of te^achers identified 

• Topics for research identified 

• Findings of the higher education community on the solution of problems made available 
at the local level 
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Disadvantages; 



• Difficulty of coping with problems not normally experienced by teacher educators 

• Special provisions for teacher educators' time required 



Alternative 5: Schedule state staff follow-up visit to local school to receive reports on plans for" 
implementing recommendations and for becoming aware of state assistance needed. 

' * 

Advantages: ^ ^ 

% \ 

• Consideration of the recommendations and a plan for appropriate actions promoted • 

• Awareness of the need for assistance by the local school made clear to state personnel 

• Checklist for future review and work with school provided 

• Corrlmunication between school administration and teachers and between school/ 
and state department provided , 

Disadvantages: 

• Major effort on the oart of state stalf required 

« Potential for tension between local school and state staff increased 



Alternative 6: Have local plan and application for programs contain information on previous 
program recommendations and their disposition. 



• Feedback provided to evaluators 

• Information provided to decision-makers on response to recommerldations for 
program improvement 

« Carefully considered recommendations and program improvement result 
Disadvantages: 

• Additional paper work 

• Possible interpretation of plan requirements as interference 



Advantiiges: 
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Recommendations and Observations 

Because of the wide variety of recommendations that usually are made to schools and 
programs, a combination of all the sefvices of support for schools should be considered. The stdte 
staff hiembers, who are most closely in touch with local programs and who have the subject- 
matter expertise, should hu\/e major responsibility fbr.assisting schools and programs in carrying 
out the recommendations of the evaluation team. However, since many of the recommendations 
fall into areas not directly related to technical subject matter, other personnel must be available 
to assist when called upon by the supervisor. Curriculum specialists an^ specialists in finance, 
special reeds, counseling, and administration should be available to schools needing that type of 
assistance. In instances where problems common to many schools or teachers are identified, one 
.of the teacher-education institutions could assist through ir<sen/ice training. 

The^te evaluation coordinator and staff should be given responsibility for seeing that the 
recohs^na^tions made by the program review teams are communicated to the most appropriate 
departm^t or individual. Those in charge of the evaluation system should follow through to be 
sure that actk^i is taken to provide the needed assist^ince- ■ '■■ 

Local sbhool a«^^Tlinistration must make certain commitments to program improvements. If. 
both school and state staffs formally accept their mutual responsibility for making program > 
improvements through the local application and plan, the impact of evaluation js assured. 



if* 
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Problem 4: To assure communication of evaluation fii;idings and results 

Those directing tho evaluation effort need feedback on what recommendations have been 
implemented and the results of the actions taken,. If service to schools is to Improve through the 
evaluation effort, the results of the recommendations made by its teams must be known. If no 

. improvement has resulted from,impilernentation of recommendations, then changes must be made 

Tn the evaluation teams and recommendations. 

The evaluation system needs proof that it is effecting changes that result in improved voca- 
tional training for students. Like all other parts of the organization, evaluation needs evidence that 
it is reaching its objectives, and feedback on improvements recommended and implemented is an 
important part of that evidence. Local school staff and students also need to become aware bf 
improvements made as a result of their participation in the evaluation exercise. 

It is also important to know about the disposition of the recommendations and the assistance 
given because such feedback information can be useful in making state level decisions. By compiling 
statewide information, the evaluation units are in a position to recomnriend where expenditure of 
support funds may be expected to produce, the best results. Administrators face many demands for 
funds. Information identifying the most common needs and most effective treatment will be of 
great assistance in addressing those requests. 

Alternative 1 : Give state staff working with local programs tho responsibility for reporting to those 
who need to know (evaluation unit, planning unit, and state administration) what each program 
and school has done to implement each recommendation. 

Advantages: 

• Major responsibility for working with schools on recommendations 
appropriately assigned 

• Follow-up and analysis of results stimulated 

9 Learning experiences provided for supervisory staff 
« Guide to most acute school problems provided 

Disadvantages: 

• Lack of time on the part of supervisors to make comprehensive reports 

• Difficulty of reporting across organizational-unit lines 



Alternative 2: Require schools to report in each year's local plan and program application the 
evaluation recommendations that have been implemented. 

Advantages: 

• Joint.work between teachers and administrators encouraged 

• Increased probability that local administrators will act on recommendations 

• School se'f-analysis encouraged 

• Guide provided for deterrnining which programs to continue, expand, or terminate 

14 
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Disadvantages: 

• Reluctance of schools to report failures 

• Lack of agreement as to what respommendations should be implemented 

• Limited resourcds a deterrent factoi'^ " 



Recommendations and Observations 

B^th of the alternatives have r erit as means of assuring that those who need to know will be 
fully informea of the results of evaluation. This reporting also provides an added stimulus to 
schools to make the improvements recommended by visiting teams. ' 

I 

Figure 1 (p 9} illustrates how the flow of assistance and information could be organized 
The process mvolves the foHowing steps: " « ^'yamzea. 

t 

1, Recommendations for program improvement flow from the evaluation system to the local 
school, the state subject-matter specialists (supervisors), and the support-services staff. 

>2. Assistance to the local school m implementation of recommendations and observations of 
results is provided by supervisors and support staff, 

3. Feedback on recommendations implemented and results achieved is transmitted from super- 
visors and support staff to the eyaluation ui^it. 

If any part of this communication network fails, the entire process becomes less effective It is the 
responsibility of the state administration to see that this or a similar communication process 
Actually occurs. 
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Problem 5: To increase communication and cooperation between evaluation and planning and 
between evaluation end the management informjtion system (MIS) 

Communication within an organization does not occur automatically or by accident. This may 
yvell be the most cHtical and most difficult job of the admlnistrator-stimulating communication. 
It is especially important that evaluati6n and planning make known to the MIS what information , 
is needed and when it is needed, and it is equally important that MIS respond to these requests for 
ifjformation. This is also true for the request and information flow between planning and evaluation 
as Illustrated in Figure 2. This ful', flow of information will avoid duplication of effort and assure 
that assistance and support are availabUi at the time and l<i the form needed. 

It must be kept in mind that organizational units do not communicate or cooperate - 
people do. Cobfimunication seems to thrive where there is mutual respect, where there is^an absence 
of fear or jealousy, where each party has a cbntribution to make, where each acts in a supportive 
manner, and where there is mutual acceptance of the objectives to be reached. Beyond this, 
communication can be stimulated by physical arrangements thee put individuals in close proximity, 
by example and encouragement from the leadership, and by rec'ognition and reward for effective 
communication! On the oth\?r hand, where there is fear or intense competition, where exclusive 
rights to information are alLwed and encouraged, where a lack of dbmmunication is accepted or 
practiced by top administration, where trust is lost, where individuals are not mutually supportive 
and contributive, and where doubts exist as to the accuracy or reliability of parts of the organization, 
people be ;ome protective and communication ceases. 

♦<* 

Alternative 1: Allow the informal process to work under the encouragement and support of the 
state administration. 

Advantages: 

« Relaxed atmosphere and lack of threat predominant 
• Enthusiasm for the working process increased 



Disadvantages: 




♦ Lack of communication not always obvious to the administration 

♦ Possibility of inadvertent failure to communicate needed information 

♦ Possibility of information flow in one direction only 



Alternative 2: Have the state administration set very formal timelines and strict responsibility for 
communication and response. This alternative is illustrated in Figure 2.* 

Advantages: 

• Some communication between parties assured 

• Administration assured of information exchange 

• Available record of requests made and responses given 
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Disadvaiitages: • . 

• PossibiJity d1 stress and resentment on part of staff 

• Only mininnal communication, that vt/hicHf satisii^js recfuirements, may be achieved 

Alternative 3: Conduct an organizational rearrangement in which these responsibilities are placed 
under one administrative head » 

Advantages; 

• Responsibility fixed for s'jeing that communication is effective 

• Communication unhindered by organizational lines / 

Disadvantages: . . 

• Move with?r^ organization usually unpopular with those affected 

• Reorganization ineffective in assuring communication 

Recommendations and Observations 

The third alternative is not acceptable to many staff members within. the vocational organi- 
zation. The informal process .{Alternative 1 ) should be tried first with strong encouragement from 
administration. If this is not effective, the formal means of communTcating should be initiated. 

Results of the informal phase may be measured by investigating the extent to which the 
information system is award of the needs and time-lines of evaluation and the extent to which those 
needs are met. The relations between evaluation and planning can be judged in the same way. If it 
\p determined that needs are not known or are not being met, Alternative 2 should be initiated. - 
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Problem 6: To evaluate services and programs for special populations 



How can the state 3</afuation system provide leaders .(stato and federal) with information on 
how effectively the special needs of various individuals are baing met-the needs or worr,iri, 
minority groups, the aisadvantaged, the handicapped, &;id those with only limited ability to use 
the English language? 1 here is need for special services designed to enable such individuals to 
succeed in a regular vocational program.* P^^ery state faces the problem of {U identifyintj individuals 
who have special needs, (2). determining whflt needs they have, (3) planning programs or services 
to meet those needs, and (4) determining the effectiveness of designed progral.is. 

Four major elements of this problem that evaluation units, with the assistance of others, mubt 
study are access, participation, process, and outcomes, (See matrix p, 23.) It must be ie^neii to 
what extent educatiorial, social, and physical barriers have been eliminated and the extent of the 
resuTtant participation of special popu'ations in vocational programs. Evaluation units must be 
aware of the special services being provided to assist the individuals in being successful.. How effec- 
tive the special activities have been in reaching the outcome objectives set for all students must also 
be determined. . 

For practical purposes it seems essential to integrate the special-needs evaluation system into 
the regular ongoing process of evaluation. Because of the limited resources available for evaluation, 
\ in terhis ofbotti dollars and staff time, a separate evaluation system for these groups simply is not 
feasible. The alternatives presented below provide some ideas for accomplishing effective evaluation 
of programs provided for speciai, as well as regular, students. • . " ' . 



^AKernatlve 1: The individual student accounting system 'should be able to identify individuals in 
, .J^^ special categories upon enrollment, follow them through the educational process,. and determine 
. the outcomes of that process. 

. ^. 

Ai^vantages: 

/ • Identification of special population individtjls by the general data-collecting system 
J ♦ Avoidance of duplication 

• Useful information provided for other functions (administration, planning, reporting, etc.) 

Disadvantages: 

• Need for close working relationship between evaluation and the Information system 

• Expenses of individual student accounting system 



Alternative 2: Local applications for specie! funds require a very specific description of the 
additional services to be provided in order to enable individuals of special populations to participate 
in programs successfully. 

Advantages: 

• Clear identification of services provided 
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• Provision for information check during team and staff visits 

• School personnel encouraged to think in terms of special assistance to individuals 
with special needs 

• Realistic accountability 
Disadvantages; 

• Special training of local teachers and administrators required 

• Difficulty in achieving a close working relationship with all personnel involved: 
the special needs unit the evaluation unit, and those .who approve expenditures 
for special funds 

Alternative 3: Program review, by staff and/or team, should provide information on the special 
services actually beino provided by the program and school and, where appropriate, some estimate 
of the effectiveness of these services. Efforts at recruitment and barrier removdl ^hould also be a 
part of program review. 

Advantages: 

• Information from program application verified and supplemented 

• Knowledge of services and results av^.quired by team members 

• Services and results observed and reported 

• Schools encouraged to expand and improve special services 

Disadvantages: 

• Evaluation team's work expanded 

• Need for representation of special-needs groups on evaluation team 

Alters ;ve 4: Data and information should be analyzed in order to tell the state - 

a. the extent to which special populations are being served; 

b. what supplemental or additional services are being provided; 

c. the comparative success of orograms in producing desirable outcomes; and 

d. the relationship between services provided and outcomes. 

Advantages: 

^ Assistance in making state decisions provided 

• Guides to most effective services for certain needs provided 

• Awareness of compliance or nonconripliance with the law created 
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Disadvantage: 

• Additional effort required 

Recommendations and Observations 

One source of assistance and expertise available to the state vocational education department 
is the special education section in the education department The relationship with this section 
needs to be strengthened through a conscious effort to communicate and cooperate. Evaluation 
should determine not only how well special needs individuals are being served at the»local level but 
also how effectively ail sources Of support are being combined to solve the problems at the state 
level. 

* 

Each state should review its present evaluation system for the purpose of identifying the gaps 
which must be filled if an effective evaluation for special populations k to result Can the informa- 
tion system show the level of participation of special populations? Does the program review instru- 
ment lead the team or supervisor to determine efforts to increase program accessibility and to 
{provide special services? Can the outcome for these special populations be accessed through the 
foflow-up or some other process? 

, Effective evaluation of special needs programs continues to be a mujor problem for vocational 
educators and evaluators. The present evaluation system can be the solution, provided the system 
has enough flexibility to respond to special heeds. The matrix displayed on the next page gives an 
overview of the four;najor elements (access, process, participation, and outcomes) that must be 
measured if a state is to be able to determine its effectiveness in serving individuals in special groups. 
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Cause Effectiveness Show 


ACCESS 


PARTICIPATION 


1.0 Facilitv Fartors 


1 .Lr tnroiimenis 


1 1 Arrhitprturp anH PniiinmPnt 


-H. 1 Disadvantaged 


1.2 SitP 1 nrstinn /nnn<-Hi';prinnlnatnr\/\ 


1 ,z nandicappeu 


1.3 SitG SBlGCtion (non-rli<»rriminatnrv) 


\,o iviinoriiies 


1.4 Modification of Physical Plant 


1 4 1 imItpH Pnnlich.^naobirtn 


1.5 Comparable Facilities 


1.5 Sex Designation 


1.6 Housing Opportunities ■ 


1.6 Age (elderly) 


1.7 Topographical Factors 


1,7 Instructional Setting 


2.0 Education-jl Factors 


I 


2.1 Recruitment 




2.2 Admission Criteria 




z.o rrogram Offerings 




2.4 Altitudinal Barriers 




o.u oocietai hactors 




o. 1 Miiiiuainai oariiers 




3.2 Behavioral Barriers 




3.3 Economic Barriers 




PROCESS 


UU 1 LUMbo 


1.0 "Additional Services" 


1.0 Student Achievement 


1.1 Administration Related 


1J Skills 


1.2 Guidance and Counseling Related 


1.2 Acquisitions in the Affective 


1.3 Instruction Related 


Domain 


1.4 Placement Related 


2 0 Successful Proaram Pnmnlptinn 




2.1 Grades 11 and 12 




2.2 Postsecondary 




2.3 Adult 




■ ^ 2.4 Apprenticeship 




3 0 SLlcce<»<5ful Plappmpnt 




3.1 Employed 




3 2 IJnpmnlnvpH 




3.3 Pursuing Additional Education 




3.4 Status Unknown 




4.0 Successful Employment Over Time 




4.1 Duration 




4.2 Promotions 




4.3 Salary Increases 




4.4 Reaction to Training 



Figure 3. Evaluation of programs and services for special populations 
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Problem 7: To piovide evaluation services when only limited resources are available 

A question uppermost in the minds of evaluation staff and administration is how the state can 
provide evaluation services that will meet,the ne&ih and the requirements of the legislation. When 
resource's are limited, an alternative means of reviewihg all state programs must be sought. 

'Though the evaluation process is viewed by legislators, administrators, and the public as 
holding great potential for program renovation and improvement', evaluators and teacherfi probably 
see the potential of evaluation in a more realistic. light. Outcomes expected of an evaluation system 
are to provide the following: 

1. Stimulus for local program improvement 

2. Accountability information ^ 

3. A guide for allocation of resources 

4. A guide for determining needs of programs ' ^ 

5. Assistance in setting objectives for programs which meet the ne^ds of students, 
employers, and society 

6. A guide for determining accessibility of programs 

/ 

7. A guide for determining effectiveness of programs for regular and special students 

8. An indication of compliance or non-compliance with federal and state legislation 

9. An identification of teacher inservice needs 

10., Information on training needs of administrators r 

11. Feedback to teacher educators 

12. A guide for curriculum revision 

13. identification of support services needed 

14. information on placement and demand for vocational education graduates 

15. Support for teacher or adririinistrator recommendations 

16. Motivation for teachers to improve 

17. A guide to program termination or revision 

18. Justification of funds spent 

19. A guide to identification of research needs 

The Education Amendments of 1976 specify that all vocational programs must be evaluated over 
a five-year period of time. The law also gives the criteria upon which that evaluation shall be based. 

The disproportionate allocation of expectations and resources creates an absolute necessity 
foi efficiency and innovation in seeking alternative ways gf meeting objectives. 
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Alternative 1 : Divert resources frorn other uses to evaluation. . 
Advantages: 

• Strengthened evaluac.'civ cV.ort 

• A demonstration of state interest in evaluation and consequent program improvement 
e State needs and federal evaluation requirements met 

Disadvantages: 

• Weakening of other activities of the' department 

• Possible resentment toward evaluation 

• Uncertainty of immediate evaluation provided to programs with greatest needs 

Alternative 2: Use a random sample of programs and teachers in order to project to the total 
population. 

Advantages: 

• Equal opportunity for all programs to be evaluated 

• Money and time saved 

Disadvantages: 

• Possibility that programs with greatest need would not be evaluated first 

• Possibility that programs with least need would be evaluated first 

• Reflection of only the average rather than the specific needs of programs 

Alternative 3: Evaluate on the basis of outcome data, and provide team visits for programs needing 
help most. 

Advantages: 

^ Evaluation provided to those programs needing it most 

• Requirements of federal legislation satisfied 

• Possible reduction of effort needed to evaluate and to assist in program improvement 
Disadvantages: 

• Ntcessity to rethink, reorganize, and reschedule the evaluation effort 

• Lack of coverage for all programs within the five-year cycle 

• Need for an effective data system imperative 
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Aiterhatfye 4: Allow local schools to conduct their own outside evaluation with guidelines and 
assistance from the state, 

Advantages: 

• Reductior] of requirements on state staff and budget 

• Increased local interest derived from increased local school responsibility r ^ 

• Increased impact on local school 

Disadvantacjes: 

• Need for the training ai»u u^Jstance of local administrators and staff 

• Potential for "buddy" system to develop between schools 

• Resistance of ^^chools needing evaluation most to participate in evaluation 



Recommendation? and Observations 

Alternative 1, diverting resources from other uses, holds little promise of implementation in 
most states. Alternative 2, sampling, would provide some^tate-wide information, but would do 
very little in provic|ing guides for program improvement except in those areas visited., 

Another view should be taken of evaluation in relation to the law as outlined in Alternative 3. 
If evaluation means determining the extent to which programs have reached certain student-outcome 
objectives, then the process becomes a matter of collecting the kinds of data that will allow this 
determination to be made. If the state staff, working with local vocational educators and others, 
can specify acceptable levels of outcomes for programs, the data can be accumulated to determine 
which programs are effective and which are not. This also requires acceptance of the fact that 
so-called "evaluation teams" do not actually evaluate but rather perform the very important func- 
tion of recommending what programs should do to improve (i.e., better meet the established outcome 
objectives). 

With these two concepts accepted. It is possible for the state, through its diita analysis, to 
identify those programs that need to be evaluated. The programs with the poorest performance 
record, although they may comprise only 20 percent of total programs, are the ones most in need 
of assistance. 

Those programs performing successfully (meeting outcome objectives) do not need team 
reviewers, at least not so much as other programs. By this alternative, the number of programs to 
be reviewed by a team visit would be reduced, with those programs needing assistance most being 
helped .first. This system also appears to comply with requirements of the law that programs be 
evaluated every five years. As a matter of fact, with the data system providing information to the 
evaluation unit, evaluation of every program every year is possible. 

The effort can be further reduced if review teams are asked to review onlv those processes 
affecting the outcomes found to be low. For example, if the percentage of completers available for 
placement has been identified as an important outcome, it may be important to look at the student 
selection and coi^seling process rather than at the quality of instruction. On the oiher hand, if the 
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data show a low rate of success in placement on jobs, the quality (if instruction would be a very 
innportant part of the process to review. Fully developed, this sysfenfi could greatly reduce the 
number of programs to be reviewed and decrease the area of process to be studied. 

Allowing jchools to conduct their own evaluations may have merit in some instances. This 
would not be self-evaluation but rather a fully acceptable outside assessment of effectiveness with 
the state providing guidelines and instrumentation td" assure uniformity and assisting in other ways 
^ as needed. 

Experimentation with these and other approaches should be tried by various states with 
results reported to the total evaluation community. 
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Problem 8: To evaluate the state vocational delivery system 

All states '^re engaged in some type of evaluation of local programs. This assessment consists 
mainly of answering th'e question, "Are we doing things right?". Few states are concentrating on 
evaluation of the state program of vocational education and attempting to determine "Are we doing 
the right things?" The accountability report specified by, the Congress, at least to some extent, asks 
this latter question of the states. ,' " ■ 

*■ 

Each state must ask Itself these difficult questions: 

1. To what extent are vve meeting the manpower needs of the States? 

2. Are programs accessible to all who can profit from training? 

3. Are we providing training for the occupations most critically needed? 

4. Are we providing training for occupations with adequate remuneration for 
training and desirable expectations of Advancement? 

5. Is there a reasonable expectation of employment for every student enrolled 
in vocational training? * 

6. Do our programs provide opportunities for retraining and upgrading? 

These and many other questions make up the score sheet against whidh state programs are 
being judged. However, a mere summation of the evaluations of local programs will not provide a " 
sufficient estimate of the effectiveness of vocational education in.the state. Alternatives as illus- 
trated below must be explored to identify areas of needed improvement In the state program. 

Alternative 1: State use the accountability report to judge its effectiveness. 
Advantages: 

, • Whether or hot the state has carrier^ out its projected program determined 

• Requirements of federal legislation complied with 

• Accountability at both state and federal level provided 
Disadvantages: ^ 

• No assurance that what was done was what should, have been done 

• Emphasis on fiscal matters and processes rather than on outcomes 
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Alternative 2: State deyelpps, by whatever means appropriate, a series of student outcome 
objectives to be answered by statistical data. ^ 

Advantages: . ■ • 

• Oi/tcomes rather than processes emphasized 

• /\ guide provided for data collection andanalysis 

• Effort and support for achieving objectives consolidated 
Disadvantages: 

• Changes in the data system required 

• Reluctance of the state to be accountable for student outcome objectives 

• Overly selective admission policies possible > , 

Alternative s: Depend upon the State Advisory Council for Vocational Education (SACVE) to 
evaluate the state program. 

Advantaged: 

• Requirements of federal legislation complie^ith 

• Involvement of many individuals outside education achieved 

• State encouraged to identify and work toward specific measurable objectives 
Disadvantages: 

\ 

• Insufficient knowledge of vocational education on the part of some Advisory Council 
members 

• Possible bias on the part of individual members of SACVE 

• SAQVE members not trained in evaluation 

Alternative 4: Bring in experts from outside the state.to evaluate the state vocational program. 
Advantages: 

• People who are knowledgeable in the area of vocational education utilized 

• Credibility of evaluation assured 

• Outside-state experience available 
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Disadvantages: * 

» ® Costly in terms of money and time 

• Inappropriate recortimendations created by differences between states 

i ■ 

• Possible need for titaining visiting teams 

Alternative 5: Determine the needs of the state and its vocational education role through survey, 
questionnaire, conference^ committee, etc., with evaluation based on needs met. 

Advantages: v . h 

• Direction for vocational e^ducation provided r: ^ 

• Evidence of the contribution of vocational education to'the state'provided 

* 

• Mahy other people involved In setting priorities ■ 
Disadvantages: ' • / 

• Unrealistic expectations of vocational education held by many people 

• Fixtensive Investigation and data collection required 

Recommendations and Observations 

States should consider the formation of a consortium of states including vocational staff and 
SACVE members to do the following: 

rr-> 0 

1. Develop que^tions.to be, answered " v 

2. Identity methods of obtaining dajta needed to answer questions 

3. Exchange personnel to review operations at,.the state level and to make 
recommendations 

Alternative 5, determining state needs through surveys, could be expanded to include an 
on-site visit by clients served (local educators and administrators). This, in combination with self- 
evaluation by the state staff, could provide useful insights into needed changes, 

Stcltes have a variety df evaU^ative information upon which to base decisions. A systematic 
approach to planning and evaluaiibn that organizes the input from SACVE, the Office of Education 
MERC-Q reviews,iaccountability rroorts, state evaluation reviews, and revfews of local programs 
can form the basis for long-range improvement of the state vocational delivery system. 
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Problem 9: To systematize all evaluation efforts 

' -f 

jyiany states have parts of evaluation system that are operating efficiently, but the effec- 
tiveness of evaluation is, In many cases, limited because the parts operate in isolation-dependent 
on, yet almost unaware of, what other divisions of the department are doing in evaluation. 

This absence of a system that would focus each of the §pparate parts into an organized effort 
can produce very detrimental results: duplication, confusion, and conflict. The purpose of varied 
input Into evaluation is to present different views of how programs should perform and hovv they 
can be Improved, advantages that may be lost if local edudators become confused or perceive 
different evaluators as giving opposing recommendations. 

These adverse effects, however, cap be avoided by a systematic approach that organizes effort 
so that everyone is aware of the role and responsibil'ty of each part. Each of the entities is best 
suited to specific parts of the evaluation effort-data collection, program review, technical assistance, 
etc. Suggestions as to how a state should proceed to'develop guidelines that specify role and respon- 
sibility and outline^rrangemenl;s for fitting the parts together are made in the following alternatives. 

Alternative 1: Call a conference of representatives of all grdups involved in evaluation to clarify 
roles and responsibilities. 

, Advantages: 

• * 

• The role and responsibility of each group made clear 

• Increasing efficiency and impact produced by joint input 

• Improved morale of staff involved in evaluation 
Disadvantages: 

• Systematic approach still not ensured' 

^ Administrative decisions still required to settle differences 



Alternative 2: Form an oversight committee to direct and monitor activities through the evaluation 
coordinator 

Advantages: 

• Problem viewed objectively 

• Responsibility for decisions placed outside the realm of the affected parties 

• Changes made possible without concern for "turfmanship" 

Disadvar^iages: 

• Possibility of slow decisions resulting in delayed action 

• Possibility of excessive time spent in reporting to the committee 

• A systematic direction to evaluation not guaranteed 
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Alternative 3: Use a systems expert (internal or external) to design a systeTti. " 
Advantages; 

\ 

• Basis for a system created . 

• Impact of evaluation increased 

• Roles and responsibilities clarified \ ' . 

• Decision reached objectively 

Disadvantages: 

• Dissatisfaction with tlie decision on the part of some individuals 

Multiple problems created if wrong individual is selected to design the system 

Alternative 4: Make sure that there is uniformity of instruments and evaluation guides and that 
each individual or group thoroughly understands its assigned role. ^ ^ 

Advantages: 

• Increased stabilization of procedures and outcomes . ' 

• Improved communication between individuals and groups 

• Reduction of duplication and misunderstanding 

Disadvantages: 

• Additional staff time required • , 

•" Reluctance of some individuals to see their role differently and to charige - 



Recommendations and Observations t 

Installing a system is difficult to do in a piecemeal way, and to be effective its implementation 
must go "all the way." There s-e steps, however, that can facilitate the accomplishment of this 
objective. A person who understands systems is essential. This person/ working with a group of 
individual^ involved in all phases of evaluation under the leadership of the coordinator of evaluation, 
should design the basic system, deciding who should do what. After administrative review, a work- 
shop should be scheduled for all individuals who are- interested and involved to suggest and discuss 
revisions. With revisions complete, the final report on the system can be made and training initiated 
to implement the system. 
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Problem 10: To acquire employer follow-up data 

It is important that decision makers (at state and local levels) and teachers know how 
employers view the training acquired by recent vocational completers. Employer viewpoint, along 
with a number of other inputs, pro\?ides valuable information as to how. well programs are accom- 
plishing student outcome objectives and what changes need to be made in programs. 

This information may be difficult to obtain if employers, considering it an rnfringement on 
their tinrie, refuse to, respond. Also, there may be concern for the confidentiality of 4hls information 
in the minds of employers. In larger businesses, the inquiry may go to a person v^iith no knowledge 
of the employee.' Another factor is the considerable expense of collecting adequate and reliable data. 

In spite of these drfficulties, it is importarft that ways to gather thjs Information be explored. 
First, the EducatJon AmendmePjts of 1976 mandatejhat employpr reaction to training be used in 
evaluating programs. Second, this informationjs valuaWe in accorrkplishing pVogram improvement 
Third, it may b^B valuable in convincing o^jhers of;^^^ education. Fourth, this 

survey^can be a factor in strengthening relationships between vocational education and the business 
community. • ' ^ ^ . \ 

^ ' Two m'ajor decisibns must be made by those charged with responjjibility for collecting infor- 
mation from employers: (1) the best method for securing the initial am continuing cooperation of 
employers and 12) the selection of the best system to ensure the right tb privacy. , 

With employer reaction to training a part of evaluation, it may be tniiteniployers o*f students 
from a particular school or program would need to be surveyed only every filFthhy^r^ 

Problem 10A: To secure bontinuing cooperation of employers 

Since employer perceptions of the quality of training are to be a continuing part of the state 
evaluation system,, a method must be devised that will assure an acceptable level of response over a 
number of years. A few states have received up to 75 percent response from employers. Whether 
or not this level of response can be maintained remains to be seen. 

In observing a number of employer follow-up efforts, the author has observe^ certain ste(3s 
that should be considered in establishing an employer follow-up: 

1. Include only questions needed to obtain essential information. 

2. Personalize tne contact: ask a certain employer about a certain employee. 
.3. Explain how information will be used for the employer's benefit. 

4. Involve employers in planning a prpcedure. 

5. Allay confidentiality fears. 

6. Obtain support of employer groups. 
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Alternative Information-Collection Procedures 
Alternative 1: "^ail survey instruments.^ 
Advantages: 

• Method of administration easy and relatively inexpensive 

• Analysis of response easy • 

• Little trouble Imposed on employers 

• Evidence given that vocational educators are attempting to .meet the needs of employers 
Disadvantages: - v ^ 

• Little personal involvement of educator and employer provided 

• Possible doubt that the inquiry gets to the person nriost knowledgeable about the 
former students' worl<; ' 

• Possible resentment of the inquiry on the part of the employer 

• Misundeistandino of question or response possible 

Alternative 2; Have teacher or student interview employer personally. 
Advantages: * . . 

• Personal involvement of teacher or student with employers made possible 

• Educational experience provided for students 

• Direct feedback to teachers made ^bssible 
0 In-depth information obtained 

Disadvantages: 

• Many additional hours required 

• Difficulty of analyzing data gathered 

• Confidentiality questions raised and rights to privacy possibly violated 

• Another individual bias element added 

/ 

Alternative 3: Secure cooperation of employer organization such as the Chamber of Commerce. 
Advantages: 

• Increased interest and support of vocational education on the part of employer 
organizations 
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• Probable increase in employer respohse 

• Increased interaction between educators and employers 

Disadvantages: ^ 

• Possible difficulty in obtaining cooperation oif employer organization 

i 

• Improbability of securing information unless employer holds membership in the 
particular employer organization 



Alternative 4: Conduct a telephone interview. y 
Advantages: 

• Response numbers increased 

• Minimum amount of employer's time required 

«> Greater insight into response gained than that obtained from a written instrument 
Disadvantages: 

• Difficulty of contacting person with most information about employee 

• Difficulty of securing the employer's confidence 

• Trained interviewer and standardized questions required 



Recommendations and Observations 

When a follow-up of students is conducted, it is important to obtain the correct name^nd 
address of the company where they are employed as well as the name of the person most knowl- 
edgeable about their work, usually an immediate supervisor. Contacting this person should insure an 
informed response if a response is received. 

In determining the best method for securing information, a state should fully explore each of 
the alternative methods mentioned here and any others which may be presented. The person 
responsible for conducting the employer follow-up should first obtain the input of a few employers 
to help in determining methods and procedures. It is important that various ways of successfully 
carrying out this task be explored and that the methods be shared among the states. 



Problem 10R: To assure the confidentiality of employer follow-up 

The st'Jte is asked to obtain perceptions of employers on the comparative quality of perfor- 
mance by recent participants in vocational training. Employers will be asked to make judgments 
relative to the employee's technical knowledge, work attitude, work quality, and overall training. If 
t^ ' is perceived as an invasion of privacy by either the employer or the employee, the response rate 
may be drastically affected. The Vocational Educatiori^Data System has proposed that the employee 
be identified to the employer and that the identification be removed before the response is returned. 
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An alternative is to explain to the former student Ir. the Vollow-up Instrument that the state wants, 
to contact the employer and to ask for the student's permission. States have received up to 80 per- 
cent approval of the students responding. 

Alternative 1: Use the VEDS "tear off" method. 
Advantages: 

• ' Federal requirements complied with 

• Analysi's possible relative to program, school, and type student 

• Permission of student nci. required 

Disadvantages: 

Determination qf which individual has responded impossible 

• Anonymity impossible if employer fails to remove identification 

• Confidentiality not maintained if information is collected on one, employee 
from one employer 

• Possible feeling of insecurity on the part of employers 

Alternative 2: Request former student's permission to contact employer. 
Advantages: 

• Employer acceptance increased 

• Former students informed cf the state's intentions 

• Questions of confidentiality satisfied 

Disadvantage ■ 

• Possibility of reduction in student response rate 

• Little likelihood that employees with problems will grant permission to contact employers 

• Other approaches eliminated if the student refuses permission to contact employer 

Recommendat'ons and Observations 

Those working in the public sector have become very conscious of privacy laws, and employers 
are no exception. If they even suspect the possibility of exposing themselves or their company to 
charges of violating the privacy rights of their employees, they will not respond. „ 

Any procedure adopted should have a legal leview before implementation. The alternatives 
suggested rieal onlv with a written instrument procedure. How the oersonal inten/iew with employers 
may affect confidentiality should be thoroughly investigated before this method is adopted. 
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Problem 11: To increase the effective? use of evaluation information 

This problem encompasses the total evaluation system and needs to be treated as a whole 
- rather than as separate parts if continuity is to be established. 

The types of evaluation information that may be available in most state systems are as follows: 

1, Student follow-up 

a. Information relative to employment, non-employ m-^nt, and further schooling 

b. Student reaction ^to training as it c{j|p||res with the work situation 

2, Employer follow-up 

a Level of satisfaction wjth performahce of former vocational students 

b. Comparison of former vocational student with recent non-vocational employees on 
several specific attrib jtes 

3, Additional data 

a. Facilities - size and condition 

b. Equipment - type and quality 

c. Personnel — experience and education 

d. Expenditui^es -Xs^i program and by student 

4, Results of self-evaluation by local school and programs 

5, Recommendations of visiting teams for improvement 

The use made of this information is the real determinant of the benefits of an evaluation 
system. Evaluation data and information properly used can significantly affect the decisions made 
at both the state and local level. 

This problem actually encompasses both the use to be made of the information and the form 
and procedures to be used in presenting the evaluation findings. Alternatives will be considered for 
three questions related to ^he overall problem: (1) how the information may be analyzed and 
packaged for use by local schools or programs; (2) the kinds of decisions evaluation information 
may be used for; and (3) increasing the use of evaluation information in planning. 

Problem 11A: To analyze and package data for use by the local school or program 

Alternative 1 : Send each program and school data on its own students and programs with no 
comparisons. 
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Advantages: 

• Avoidance of unfavorable comparison between schools and programs 

• Less threat to teachers and admini^rators 

• Minimum data handling 

• Comparisons made on the |asis of school or prograr.i performance on yearly basis 
Disadvantage: ^ 

• No opportunity for schools and programs to compare their results with those of ^ 
other schools * 

Alternative 2: Send each program and school data on its own students and programs with averaged 
state data for comparison. 

Advantages: 

• Comparison points provided 

• No identification made except thatpf self-identification to individual's school 
and program 

Disadvantages: 

• Some additional data analysis required 

\^ 

• Possibility of state averages being deceiving as a basis for comparisons because of 
differences in basic school structure 

Alternative 3: Send each program and school data on its own students with averages for programs 
in similar situations. 

Advantages: 

« Equitable basis for comparison provided 

• Relation between ability to pay and program quality possibly revealed 

• Level of administrative support indicated 

Disadvantages: 

• Extensive data treatment required 

• Difficulty in determining similarity of situations 
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Alternative 4: Calculate a "product index" or "outcome index" that combines results of attain- 
ment of. several selected outcomes into one figure for each program or each school. 

Advantages: ^ 

• One figure expressing program impact provided 

• Important outcomes identified 

• Emphasis placed on outcomes rather than processes 

i 

Disadvantages: 

• Extensive data manipulation required 

• Disagreement as to which outcomes are most important 

< 

• Iniippropriate analysis for programs having different or unique outcome objectives 

Alternative 5: Establish outcome objectives expected of each program and school and cliecK data 
against these established standards. 

Advantages: 

• Indication provided to teachers, students, parents, and community of what vocational 
education is attempting to achieve 

• Program weaknesses and areas needing assistance indicated 

• Comparisons mentioned in other alternatjves possible with this analysis 

• System applicable to different types of programs 

/ 

Discidvantages: 

• Disagreement on outcomes likely 

• Possibility of required changes in data collection and analysis methods 



Recommendations and Observations 

States should provide local schools and programs with all available data while continually 
considering how the amount, quality, and usefulness of the information can be increased. Eventually, 
states should consider Alternative 4 or 5 as a method of further analyzing student accounting and 
program data. The calculation of an outcome index (Alternative ^) involves the reaching of a con- 
sensus as to desirable outcomes, the assignment of a weight to ebch outcome, and the development 
of a formula and computer program to perform the calculation. This reduces all the expected impor- 
tant outcomes to one figure and is a method of indicating the relative effectiveness of programs in 
achieving those outcomes considered important. 
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Alternative 5 would be the measuring of a program or school performance against previously 
established outcome objectives. It Is important thav these objectives be stated In terms of measurable 
outcomes rather than processes or procedures. These objectives may be set by program and by 
school, or statewide by program. The procedures followed in setting objectives would be a state 
decision, but certainly the involvement of those affected, including local educators, would be essf/n- 
tial. Such a system would also serve as a guidf) in program review. ' 



Problem 11B: To use evaluation information 

The following are the major uses at the state level: ^ 

4 

1 . To facilitate decisions relating to expansion, continuation, or termination of programs 

2. To serve as a guide for state subject-matter specialists in effecting school/program 
improvement ^ 

3. To serve as a guide in determining fund allocation for supplemental services 

4. To indicate inseryice and preservice education needs 

5. To indicate the impact of vocational education 

6. To determine the effectiveness of state vocational delivery systems 

7. To determine the effectiveness of services for special needs groups 

The following are the major uses at the local level: 

1. To indicate changes needed in school and program 

2. To indicate the impact of vocational education 

3. To assist in decisions on fund allocation 

4. To identify programs needing.special assistance 

5. To assess effectiveness in meeting the needs of special population individuals 

Obviously, both local and state decision makers should be using the results of evaluations as 
an input for managing vocational education programs. Too often, however, these results are not 
used, partially because the manager is unaware of the potential benefits of this procedure and 
partially because evaluation findings are npt packaged in the most usable form. If decision makers 
and evaluators are made aware of the possible uses, both parties can direct their efforts toward the 
timeliness and usefulness of evaluation results and reports. It is also important to^ realize that input 
from many sources other than evaluation results should be used in decision-making. 
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Problem 11C: To increase the use of evaluation information and data in planning 

Data from the IVIanagement Information System (IV! IS) and information from evaluation can 
be important inputs for state planning. Because state administrators and planners make many 
decisions on program implementation, expansion, and termination, it is important that these 
individuals have all relevant information available and that they use it in making these decisions. 
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There may be several reasons why information or data are not used in making decisions: 

1. Decision makers (planners) may not know the information is available. 

2. The data and information may not be in the form that the planner needs. 

3. ^ The planner may not have confifience in the c'ata. , , 

4. The planner may not know how to use the data or information. 

/ 5. The information nfiay not be available when the plinner needs it. 

Most of these- reasorls for not using €valua^1on information and data focus? on lack,of commu- 
nication. How this communication can be increased is a problen) that should be of concern lo M'l 
state administrators and staff members. Communication and cooperation betvii/een MIS and planning 
seem to be well-established in most states. Concern about the flow of information between evalp- 
ators and planmers appears justified. Some acions that may help to alleviate this problem are 
outlined below. ' J 

1. Evaluation specialists should make know^to planners the information' vvhich they 
can provide. 

2. Planners should specify to evaluation -staff the'content, form, and time information 
is needed. i • 

3. Administrators should insist that information is requested anS provided. 

4. Planners should be required to specify how evaluation information is used in planning. 

One of the major functions of planning i$ thj allocation of resources. Evjaluation determines 
productivity, quality, and needs. Considering jthesie two facts indiciates thatofaluation can assist in 
determining the effectiveness of past expei;idltur€(s" (productivity and quality) and futufb allocation 
of funds (needs). 

The preface of this report gives further clues as to how cofmrnUnication can be stimulated in 
an organization. . 
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Problem 12: To measure student achievement v.* 

The 1976 Education Amendments mandate that student achievement be used as arte of the - 
criteria in evaluating vocational education programs. A survey of state evaluation cboixHnators 
Identified the measurement of Student achievement as one of the four major problerR^common to, 
the states. State directors of vocational education, when surveyed during a national conference in 
September, 191^8,. ranked this problem in the upper quartile of evaluation problem's in the states. 

The whole field of competency teaching, testing, and reporting would appear to be well suited 
to vocational education. Certainly, if we know specifically what itis we want students ta learn, and 
If we construct ways of measuring ^ind reporting thi^^learning, we have a reasonable method of 
evaluating programs. " - 

Curriculum materials are being developed that concentrate on teaching and testing the 
competency attainment of students. Many schools in several states are moving in the direction 
of more specific determination of student learning. The reporting and accumulation ol| this infor- 
mation for use in evaluating programs is an area needing development. 

Alternative 1 \ Use competency testing and reporting of student achievement at xhool/program 
level (use of competency-based curriculum). 

Advantages: ^ ' 

! 

• Testing, v^hen performed by the io(;al school, simplified , 

• Inforrfiation available for designing program changes to meet local needs 

• Measures of the effectivenes^of programs provided 

• Valid and reliabte tests being developed 

Disadvantages: 

^ • Possible differences in competency requirements between schools even in same 
program areas 

• Possible limited view of competency requirements within an occupation on the part 
. of teachers unless employers involved * 

• Possible "teaching to test" 

• Valid and reliable tests nonexistent in so/ne instructional areas 

• Extensive inservice training to ensure the reliability and validity of competency 
testing required 

• Lack of uniformity of curricula 
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Alternative 2: Use statewide pompetency tosting and recording of Student achlevemelit. 

Advantages: • . . 

» •, 

y Uniformity across the state assured 

• Schools/programs made aware of what is expected as outcomes ^- 



\ 

hers made aware of^ompetencies expected of trainees 
• Art additional criterion/for determining program effectiveness provided 



• Te^h 



Disadvantages: 

• A major effort on the part of the state required 

• Nonexistence of proven competency tests in most subject matter areas 

• Threat to local school autonomy 

• Possible danger of judging teachers solely on pupil performance 

• Problem of keeping tests updated 



Alternative 3: Base §tudent achievement on success on the job: assume students successfully 
employed achieve learning^ expected. ^ ^ 



Advantages: 



T(S procedure provided by student and teacher 



• Some of the information needed for t 
follow-up 

• Schools encouraged to emphasize importance of student follow-up 

• Provides reason, for the state to expand follow-up beyond the first year 

Disadvantages: « 

• Time required to measure job success 

• Not applicable to students who do not become employed 

• Difficulty in judging effects of outside influences on employment 

Alternative 4: H&ve students rated by skilled individuals from outside the school. 
Advantages: 

• Greater community involvement in the vocational program 
Valuable experience provided for students 

• Stimulus and aid to employment provided 
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Disadvantages: ^ 

« Difficulty of maintaining uniformity\ 

V 

• A major effort on the part of ihe state cind local schools required 

• Possible.reluctance of programs to accept floorer students 



Recommendations and Observations 



Obviously, whichever alternative is chosen, a major effort will be required-^^his is a long-range 
objective requiring from the state a considerable amount of time and resources over a period of 
several years. The National Occupational Competency Testing Institute (NOCTi) is presently in the 
process of testing and hiodifying their testing materials to be appropriate for use w^ith students. 
Those working on methods of measuring student achievement should be alert to the NOCTI and 
other testing programs under development. 

States should begin to test alternatives. Validation of curriculbim n[iaterials is a complex and 
time-consuming activity. If agreement can be reached on competences required for successful 
employment, then curricula can be constructed based on these competencies, and tests can be 
prepared and provided to determine the level of achievement of individual students. The curriculum 
labs in vocational education are a starting place for recording student achievement. If tests do, in 
fact, measure competency, and if curriculum materials are valid, this procedure should give reliable 
information upon which to begin to build baseline data on how well students are mastering the 
competencies needed for employment. 

Assistance in measuring student achievement will be provided by a National Center publication 
planned for 1980 entitled "Performance Testing: Issues Facing Vocational Education." This report 
will consider the legal, philosophical, technical, and implementation factors of performance testing 
and the implications for vocational education. 

Th^re may be many other areas of student achievement in addition to job competency that a 
state or local school may wish to measure. A project of the National Center, "Examining Vocational 
Education Outcomes and Their Correlates," has identified several possible areas, including the skill 
of communication and numerical calculation, consumer and other self-help skills, and world of work 
knowledge. Currently, project staff members are developing procedures for measuring some of 
these outcomes. ^ 
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Problem 13: To determine the effectiveness of the evaluation system 

I 

State administrators and evaluators at both the secondary and postsecondary levels have an 
interest in assessing the effectiveness of their evaluation system. Concern has been expreissed for the 
collection of information on (1) the extent to which the present system meets the) federal require- 
ments, (2) the extent to which the present system is meeting state and local needs for program 
improvement and administration, and (3) methods of improvement. 

Those in administration and thbse in evaluation seem genuinely interested in improving the 
evaluation system, and the requests for an assessment appear to stem from that interest. Adminis- 
trators are understandably anxious about compiiancei while evaluators naturally are concerned 
with the cornplex problems of making their plans operational. The following alternatives should be 
considered as possible methods of evaluation. 



Alternative 1 : Use knowledgeable persons'outside the system to evaluate the evaluation system. 
Advantages: > 

• An objective view of the system obtained 

{ ) ^ ■ 

• ^Credibility of the results to outsiders increased 

• Strong support for improvements provided 

• Possibility of extending procedures developed. as an evaluation guide for other states 
Disadvantages: 

• Additional resources required ' . 

• Time required for orientation and understanding of people unfamiliar with the system 

• Possible state preference for a continuing system, not one dependent upon outside input 

Alternative 2: Have the evaluation staff, in conjunction with the state staff and others, develop a 
system for self-analysis of strengths and weaknesses of the evaluation system. 

Advantages: 

• Continued use and improvement of the systein by the state 

• State input as well as evaluation knowledge available for developing the system 

• Development of a long-range plan for evaluation improvement made possible 

Disadvantages: 

• Time and effort of both evaluation staff and other state staff required 

• Becomes imperative to provide a very objective procedure in order to avoid dangers of 
self-interest 
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•Alternative 3: Base evaluation on judgment of users (state legislature, state agency management, 
state advisory council, local schools). 

Advantages: . 

• Usefulness to users made the primary criterion 

•, People responsible for the decisions about vocational education and evaluation involved 

• Feedback on changes needed in evaluation system provided 

Disadvantages: " • 

• Difference in the needs of the users 

• Difficulty of anticipating needs 



Recommendations and Observations 

Different states will want to use different methods for evaluation of the evaluation system. 
Alternative 3, evaluation by users, should be strongly considered. A survey of these user groups 
could provide a checklist of expectations against which to judge evaluation efforts. Administrators 
of selected state education agencies identified the following items as being important in evaluation: 

1. Consider evaluation based on other benefits of vocational education in addition 
to placement. 

2. Evaluation should go beyond compliance and should consider state needs. 

3. Evaluation should identify those programs that need help and suggest ways to 
improve them. 

4. Data, a\or,r\ with evaluation, should be used as a basis for allocation of funds. 

5. Evaluation should produce evidence that dollars spent in vocational education produce 
more payoff than dollars spent elsewhere. Payoff in terms of job training and employ- 
ability should be reported. 

6. Evaluation should provide solid evidence that vocational education makes a difference 
in jobs, pay, and upward mobility. 

7. Evaluation should provide evidence for use in eliminating or redirecting programs. 

8. Evaluation should describe program results in terms of placement. 

9. Evaluation should determine competencies achieved by students. 

10. Evaluation should measure how well teachers are doing in terms of student reactions, 
updating of course materials, contact with industry, and what students do with their 
training. 
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While thejje expectations are certainly challenging, and perhaps In some instances unrealistic; 
they provide an ideali&tic yardstick against which to measure the evaluation system. Any state 
planning to use this mdfthod should survey state and local administrators to deterrnine their expec- 
tations and needs. 

Consideration is also being given to Alternative 2, the development of criteria and procedures 
for state analysis of the evaluation systenn. Early thinking is centering on the specification of the 
essential characteristics of an effective evaluation system^ The'National Center staff has completed 
the preliminary work on identifying these essential characteristics. The input of other evaluation 
specialists at the National Center, state directors, and state evaluation specialists has bean included. 
State input and acceptance of these characteristics would be the next step In implementing this 
procedure. 

The next phase of the development of this evaluation: analysis procedure will be to v3Stablish 
measures for each of these characteristics. When these, measures have been reviewed, revised, and 
accepted, the state can proceed to gather data that, when used to measure the level of effectiveness 
of system characteristics, can give the state a profile that will indicate the strengths and needs of 
th? evaluation system. This is a long-range effort that would require extensive effort but one that, 
when perfected, should be usable for many states. 
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Problem 14: To establish a relationship between evajuption and compliance 

Lenislation relatinn to vocational ;ducation (Educfition Amendments of 1976, Civil Rights Act, 
Vocational Rehabilitation Act) makes many demands on the state and local education agencies. 
The states are held responsible for their own compliance with related laws and regulations and for 
the actions of their subrecipients; This means that the stste vocational agencies must know the 
extent to which each local program and institution is meeting the specifications of the various acts. 
The state is also responsible for the correction of any noncompliance at the local level and the 
reporting of any continuing discriminatory practices. 

There is.wide disagreement within the ranks of vocational educators and evaluators over the 
extent to yvhich evaluation should be responsible for gathering information relating to compliance. 
Oh the surface the expression "Evaluation should never b3 involved with compliance," seems 
rational until alternatives are examined. Does the statement mean that evaluation should not be 
Involved in determining the extent to which" programs are effective for those participating in them? 

Garetulj^d thoughtful examination of all possible alternatives and results is called for In this 
difficult area. * 



Alternative 1 : Give responsibility for compliance to one individual. 

Advantages: ^ ^ 

• Responsibility centered in one office, one individual 

• Compliance enforcement avoided by evaluation, MIS, subject-matter specialists, etc. 

• Source of information centered in one office 

Disadvar,udges: 

• Danger of^ isolation of compliance office 

• Danger of dual information-gathering reviews of programs for evaluation and compliance 

• Danger that other state staff members may feel no responsibility for assuring equity 



Alternative 2: Give all sections (evaluation, MIS, planning, administration) specific responsibility 
for assuring compliance. 

Advantages: 

• Avoidance of dual system 

• Inclusion of equity information as part of established systems 

• General feeling of responsibility for assuring equity among staff 

Disadvantages: 

^ Possible fear or resentment of local schools 

• Increased work load for each section 

52 
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Recommendations and Observations . 

It is reasonable and rjght that evaluatlori should strive to avoid the label of compliance 
enforcer; but by its very nature, evaluation becomes a process of meaning programs against 
sqme set of standards. The use df the term compliance is unwise in describing to schools the 
mission and purpose of evaluation; however, review of assdssibility and effectiveness of programs 
for special populationsjjoes appear to be appropriate as a part of the work of evaluation. 

Too often when one person or one part of the organization is put in charge of a particularly 
difficult program, two things are lil<ely to happen: (1) that person or office becomes isolated through 
their own or others' actions, and (2) the rest of the staff claim no responsibility for this particular 
probleni sinr.> it has been assigneo to someone else. This has already happened in some states to the 
sex equity officer and the special-education supervisor. 

Assuring equity and effectiveness' for those individuals with special needs is a job requiring the 
best efforts of everyone in vocational education. Evaluation has an important role to play in this 
lesponsibility and, with the input of specialists, should develop one evaluation system to include 
all the clients of vocational education. 
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Problem 15: Jo secure commitment to evaluation 



Evaluation, if it is to be more than just an empty gesture, must have the full support of state 
vocational administrators and staff as well as local educators. If, when asked about evaluation, 
everyone turns to the state evaluation coordinator with the comment, "It's done by that person/' 
there may be reason to have doubts about the level of commitment. As is the case with so many of 
the newer supportive services for vocational education (research, planning, MIS, sex equity, special 
education, etc."), total staff involvement is necessary if the full benefits are to be realized. 

Evaluators sho^! J b^. fully aware of their dependence upon the total staff support while 
emphasizing the contribution that evaluation can make to an improved vocational education 
system. Some alternative approaches to getting increased commitment are discussed below. 

Alternative 1: Involve all those individuals expected to participate in the planning, revision, and 
operation of tne system. 

Advantages: 

• Contribution to improved system made by several people 

• hterest and dedication increased 

• Feeling of "partnership" stimulated 

• Objectionable features removed 

Disadvantages: 

• Time required 

® Planning required 

• Elimination of some preferred points 

• Danger of misdirection 

Alternative 2: Stress the need to fulfill requirements of the legislation. 
Advantages: 

• Immediate benefits seen by administrators 
» Certain aspects of system supported 

Disadvantages: 

Becomes only comr^licince activity 

• Depencient on continuinrj leqislation 
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Aiternatjve 3: Determine the expectations of evaluation and build a system to meet them. 

\ 

Advantages: 

^ Contribution seen by local and state administrators 

• Staff recognition of potential contribution 

• Less fear experienced by teachers 

Disadvantages: 

• Dapger of developing unrealistic expectations 

• Difficulty in determining realistic expectations 



Alternative 4: Communicate fully evaluation results. 
Advantages: 

• Awareness of expectations met 

• "Pay off" of evaluation recognized 
« Administrators made aware of needs 

Disadvantages: 

; • Time taken from other work 

• Reluctance to have results fully known 

Recommendations and Observations 



Most individuals in education will agree with the idea of evaluation. The differences and 
reluctance seem to come when decisions are made as to what or who is to be evaluated and how it 
should be done. The problem of getting commitment seems to be one of demonstrating to several 
interested parties that the evaluation system will meet their expectations. Suggestions for achieving 
this are presented in the form of steps. 

Step 1 : Determine what is expected of an evaluation system. 

Activity 1 . Survey state administrators and staff on expectations. 

Activity 2. Survey local administrators and vocational teachers on expectations. 



Inability or unwillingness to state specific objectives 



Step 2: 



Determine extent to which present evaluation system is meeting expectations and 
changes needed. 




Activity 1. M^tch the list under Alternative 1 against observable or predictable outcomes of 
evaluation. 

Activity 2. Calculate additional resources needed to nneet all reasonable expectations. 
Activity 3. identify parts of evaluation not helping to meet expectations. 



Step 3: Select changes to be made in system. 

Activity 1. Calculate resources freed by eliminating non-productive parts of system. 

Activity 2. Ask administration to determine what additional resources can be allocated to achieve 
unmet expectations. 

Activity 3. Initiate changes. 

As administrators are asked to provide funds and others are asked to contribute time and 
effort/ this list may be reduced again, but in this way it can be determined what the level of com 
mitment is and what adjustments must be madu. This procedure at least lets those individuals who 
are involved say what they want and what they are willing to work for and pay for. 

After the evaluator has done all that can be done to secure' commitment of resources and 
assistance, the system must be designed to get the greatest return for the amount invested. This 
process should show what can be bought with the dollars and time invested, and the point madn 
that if more output is desired, greater i iput is required. We must face up to the fact that "wn f|f^t 
what we pay for-and no more." 
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Problom 16; To increase emphasis on outcome evaluation. 

Voccjtional educators are highly proWoriented. As skWM technicians, they hav^^enerally 
accepted the assumption that certain procedures lead to predictable outcomes. The process of 
mnning the bead or programming the computer has led to a weld of a certain strength or an 
expected treatment of the data. The inclindtibn to depend upon and attempt to perfect the process 
has carried over to work in evaluation. It is assumed that programs can be effectively evaluated by 
obsen/ing the process as an indicator of quality. As a result, square footage, window number and 
placement, dollars spent, etc. have often become the criteria for evaluation. This faith In process 
has led vocational education to the assumption that if certain things are done with students, certain 
results will automatically follow. The predictability of student outcomes is much more complex 
than forecasting the results of a procedure in welding. 

Effectiveness In evaluation depends upon .the sorting out of those factors which show or 
dem.>nstrate effectiveness and those factors which may influence effectiveness. While the number 
of contacts a teacher has with employers may appear to relate to student placement on jobs, it 
cannot be assumed that if a teacher has many contacts with employers that placement is high for 
that program. The only way to determine placement rate is to follow up completers of programs. 
In recommending what programs with low placement should do ^o improve, the program must be 
studied. It may evolve that increased employer contacts should be instituted. Thus, evaluation is 
done on the basis of the achievement of those factors which show quality (outcomes set for the • 
program) while recommendations are made on the basis of those factors which are thought to cause 
quality (process). 

Some alternatives for achieving more outcome-oriented evaluation are given below. 



Alternative 1: Development and use of an individual student accounting system which includes 
enrollment, student characteristics, program progress, placement, and follow-up 

Advantages: 

• Student information available 

• Outcome information available 

• Data colle^ion centralized 

• Student progress information available 

Disadvantages: 

• Heavy data burden 

• Expensivo 

• Lack of trust in centralized data system 

• Information difficult to secure 
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Alternative 2: Provide evaluation teams or individuals with as nnuch outeome data (from student 
accounting system) as is available. 

Advantages: 

• More complete view of program provided ' ^ ' 

• Study of process-outcome relationship possible 

• Guide to process needing examination provided 

Disadvantages: 

• Training of team members required 

• Danger of misuse or misinterpretation of data 

• Overemphasis on one outcome / , 

Alternative 3: Rank programs on basis of achievement of outcomes and provide^ssis^nce to 
less effective programs. ' ^ 

Advantages: 

• Help provided for programs with greatest need 

• Desired outcomes emphasized 

• Guide to process review provided 

• More objective evaluation given /\ 

• Knowledge of program objectives provided 

/ 

Disadvantages: 

• Embarrassment to low programs 

• Lack of agreement on outcomes 



Recommendations and Observations 

\> 

Increased emphasis on outcomes does not mean that process is to be ignored. It does mean 
that process changes will be recommended in an attempt to improve outdomes. The effectiveness 
of process changes can be judged on the basis of the effect on outcomes. This seems to be the only 
rational reason for changing the process in a program. 

Some states are identifying less effective programs on the basis of outcome data and concen- 
trating improvement efforts on these programs. The programs with acceptable levels of outcome 
achievements are less in need of assistance. The expression "if it ain't brol<;e, don't fix it" would 
seem to be appropriate in this instance. 
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CHAl»TER!ll 
SUMMARY AND CONCLUSIONS 

A number of vocational education evaluation problems and alternative solutions are discussed 
m this report. These problems reflect: (1) the personal experience of thq author in implementing 
and operatmg a state evaluation system; (2) the experience of providing technical evaluation assis- 
tance to several states; and (3) consultation, conversation, and survey of a number of state evaluation , 
coordinators. The alternative solutions presented are largely drawn from the same reservoir of ' 
experience and interaction with the added Input of the author's evaluation colleagues at the 
National Center for Research In Vocational Education. 

As one views the list of problems which presumably reflect the situations in the states, some 
conclusions begin to emerge. Some basic, underlying principles which might be concluded are 
H) evaluation depends heavily upon Information; (2) to be effective, evaluation information must 
be communicated to many audiences; and (3) communication depends upon presenting information 
which IS factual, timely, and understandable. Beyond this there are some approaches being developed 
and tried by states which show promise of solving many of the problems presently facing vocational 
education. Some mechanism should be developed which makes it possible for states to share 
experiences in evaluation. 

State and local education agencies need to keep in mind the purpose of evaluation. As stated 
by the Congress in the Education Amendments of 1976, evaluation is to be done "in order for the 
state to assist local educational agencies in operating the best possible programs of vocational 
education." If evaluation does not result in changes in programs which benefit students, the intent 
has been thwarted and a lot of resources wasted. All of those involved in the evaluation effort - 
administrators, advisory committee members, evaluators, teachers, students, team members and 
all others - can legitimately ask, "What is done differently as a result of evaluation and how will 
projram output be improved as a result of these changes?" 
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